Photography system with remote control subject designation and 

digital framing 

FIELD OF THE INVENTION 
5 The present invention relates generally to photography. 

BACKGROUND OF THE INVENTION 
A common inconvenience in consumer photography is that the photographer 
at an activity must generally tend to the camera, and thus cannot experience the 

10 activity in the way that others present might. Some cameras include a remote control 
device that can activate the camera from a distance. The photographer can position 
the camera, optionally place herself in the scene, and use the remote control to take 
photographs whenever she desires. However, this method generally gives the 
photographer little control over the composition of the photograph once the camera is 

15 positioned, and does not adapt well to changing scenes. 

The inconvenience is particularly acute in video photography. The 
videographer must typically choose between letting the camera run unattended during 
an activity, resulting in an unartful recording, or removing himself from the activity 
for the duration of the recording to tend to the camera. 

20 What is needed is a system and method for conveniently and artfully 

photographing or video recording a scene, while allowing the photographer freedom 
and flexibility. 

SUMMARY OF THE INVENTION 
25 A photography system includes a digital camera and a remote control. The 

remote confrol emits a light that the photographer can use to designate a subject to be 
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photographed by pointing the remote control at the subject, casting a light spot on the 
subject. In a preferred embodiment, the light is generated by a laser pointer included 
in the remote control. The camera selects a region to photograph from its field of 
view, based on the location or motion of the Ught spot. The camera may optionally 
select a region that is centered on the light spot. The photographer may optionally 
specify the size of the region to be selected. The camera may optionally adjust the 
size of the selected region to assist in photographic composition. The camera may 
optionally be capable of making video recordings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 depicts a system in accordance with an example embodiment of the 
invention. 

Figure 2 depicts a camera situated so that its field of view encompasses a 
relatively large area of interest. 

Figure 3 shows a close up view of a remote control in accordance with an 
example embodiment of the invention. 

Figure 4 represents an array of pixels. 

Figure 5 depicts a particular region being selected from the camera's field of 

view. 

Figure 6 illustrates how a selected photograph of part of the scene 
encompassed by the selected region may compare with a reference photograph. 

Figure 7 illustrates consecutive preliminary digital photographs and the 
detection of the position of an intermittent laser spot in accordance with an example 
embodiment of the invention. 
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Figure 8 illustrates how a camera may select a selected photograph from a 
reference photograph. 

Figure 9 shows a situation in which a selected photograph cannot be centered 
about the laser spot location. 
5 Figure 10 illustrates choosing the largest selected photograph that is centered 

on the location of the laser spot. 

Figure 1 1 illustrates an example technique for removing the effect of the laser 
spot from a video frame. 

Figure 12 illustrates a technique for selecting a region based on motions of the 
1 0 remote control. 

Figure 13 illustrates using optical zoom to improve the resolution of a selected 
photograph. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
15 Figure 1 depicts a system in accordance with an example embodiment of the 

invention, and placed in an example photographic situation where the system can be 
used to good advantage. 

Camera 100 may be placed on a tripod 101 or otherwise held substantially 
stationary. Camera 100 is directed at a scene to be photographed. Photographer 102 
20 holds a remote confrol 103, which can emit a light beam capable of casting a light 
spot on a photographic subject. 

Camera 100 may have a zoom lens or a lens with a fixed focal length. If 
camera 100 has a zoom lens, it may be configured to a relatively short focal length so 
as to give the camera a relatively wide field of view. A relatively short focal length is 
25 one that is near the shortest focal length the camera is capable of. For example, in a 
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camera with a focal length range of 6 to 18 mm, a focal length near 6 mm would be 
relatively short. As shown in Figure 2, camera 100 is situated so that the field of view 
201 of the lens encompasses a relatively large area of interest, from which regions 
may be selected to photograph. 
5 Figure 3 shows a close up view of remote control 103. In one preferred 

embodiment, remote control 103 comprises a laser pointer 104, to be used in 
designating photographic subjects. Laser pointer 104 may emit light of a wavelength 
visible to the human eye. Other embodiments within the scope of the appended 
claims may use other kinds of light sources. For example, a conventional collimating 

10 optical system may project a relatively narrow light beam, or an imaging optical 
system may project an image of a light source onto a photographic subject. 

Remote control 103 also comprises various controls operated by the 
photographer 102. For example, control 302 may cause the camera 100 to take a 
photograph. Controls 303 and 304 may cause the camera 100 start and stop the 

1 5 making of a video recording. Other controls may be present on remote control 103. 

A digital camera such as camera 100 typically uses a lens to project an image 
of a scene onto an electronic array light sensor. The electronic array light sensor 
typically comprises many light-sensitive elements sometimes called "pixels". Each 
pixel measures the brightness of light emanating from a corresponding location in the 

20 scene. The electronic array light sensor typically accumulates electrical charge in 
each pixel in proportion to the brightness of light falling on the pixel. This charge 
quantity is then measured to determine a numerical value. The numerical value is 
also often called a "pixel". The meaning of the term "pixel" is generally clear from 
the context of the reference. The set of numerical values resulting from the 

25 measurement of the charges from the pixels of the electronic array light sensor may be 



H. P. docket number 20031 1589-1 



Page 4 of 24 



collected into a numerical array. The numerical array may be called a digital image, a 
digital photograph, or sometimes simply an image or a photograph. When properly 
interpreted and displayed, the digital image reproduces the scene photographed by the 
camera. 

5 In some cases, fewer than all of the pixels on the electronic array light sensor 

need be measured to determine numerical values. For example, if a photograph of 
lower resolution than the camera is capable of is desired, or if a photograph of only a 
portion of the camera's field of view is desired, some electrical charges may be 
discarded without being measured or saved. 

10 Figure 4 represents an array of pixels 401, and may be thought of as 

representing the light-sensitive pixels on an electronic array light sensor or as 
representing corresponding elements in a digital image array. Only a few pixels are 
shown in Figure 4 for simplicity of explanation. An actual camera may have many 
thousands or millions of pixels. Many digital cameras use selective wavelength 

15 filtering on some pixels to record color information about a scene, allowing such 
cameras to produce color photographs. One of skill in the art will recognize that the 
present invention may be embodied in a camera with color capability or one without. 

In Figure 4, the entire array 401 corresponds to the entire camera field of view 
201, and in fact the size of the electronic array light sensor and the characteristics of 

20 the lens of camera 100 define the camera's field of view 201. A subarray 402 of 
pixels may be selected firom array 401 in order to select a particular region fi-om the 
field of view 201 of camera 100. In Figure 4, subarray 402 has its origin at row 3, 
column 5 of array 401, and subarray 402 is four pixels wide and three pixels high. 

Figure 5 depicts a particular region 501, possibly corresponding to subarray 

25 402, being selected fi-om the camera's field of view 201. In Figure 5, photographer 
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102 aims remote control 103 at child 502. Laser beam 503 casts laser spot 504 on 
child 502. (Laser beam 503 will normally not be visible, but is shown in Figure 5 for 
clarity of explanation.) Figure 6 illustrates how a selected photograph 601 of the part 
of the scene encompassed by the selected region 501 may compare with a reference 

5 photograph 602 of the scene encompassed by the camera's entire field of view 201. 

hi Figure 6, selected photograph 601 is framed such that laser spot 504 is 
placed in the center of selected photograph 601. If camera 100 can detect the location 
in its field of laser spot 504, and if a region size has been specified or selected, camera 
100 may accomplish such framing by selecting an appropriate subarray from 

10 reference photograph 602. This subarray selection may be called "digital framing", as 
it simulates a photographer's framing of a photograph by selecting a scene region to 
photograph from a larger choice of possible regions. The digital framing may 
typically be done by a microprocessor, digital signal processor, or other logic that is 
part of the camera elecfronics. 

15 Li one example embodiment, the location of laser spot 504 in camera field of 

view 201 may be accomplished as follows. Digital camera 100 may take a sequence 
of preliminary photographs. The sequence may be taken for the purpose of locating 
laser spot 504, for facilitating camera adjustments such as focusing or selecting a 
proper exposure, or for a combination of these. At least some of the preliminary 

20 photographs typically include the entire camera field of view 201, and may be taken 
at a resolution lower than the camera's fiiU resolution. 

la this example embodiment, laser pointer 104 on remote control 103 emits 
light only intermittently, blinking on and off repeatedly. This blinking or toggling of 
light spot 504 provides a recognizable "beacon" that the camera can distinguish from 

25 features in the scene. When laser pointer 104 is on and emitting hght at a time when a 
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preliminary photograph is taken, pixels on the camera's electronic array light sensor 
will receive light from laser spot 504, and the digital values in the resulting 
preliminary digital photograph corresponding to the location of laser spot 504 will 
indicate the presence of the light. Once laser pointer 104 has switched off and a 
5 subsequent preliminary photograph is taken, the corresponding digital values will 
reflect only the scene illumination. The location of laser spot 504 may be detected by 
comparing consecutive preliminary digital photographs and finding differences 
resulting from a change in state, the switching on or switching off, of laser pointer 
104 and resulting laser spot 504. 

10 For example. Figure 7 illustrates consecutive preliminary digital photographs 

701, 702, and 703. For simplicity of illustration, the numeric arrays are reduced in 
size as compared with a typical digital photograph. Typically, brighter scene 
locations are indicated in a digital photograph with larger digital values, and darker 
scene locations are indicated with smaller digital values, although the opposite 

15 relationship is possible. Arrays 701 and 702 are substantially identical. Differences 
in the arrays, representing changes in the digital photographs, are revealed by 
subtracting, element-by-element, array 702 from array 701. The resulting difference 
array is shown as array 704. Only a few pixels have changed numeric value between 
preliminary photographs 701 and 702, and only by small amoimts. These changes 

20 may be attributable to random noise in the camera electronics, to subject motion, or 
other effects. In order to screen insignificant changes from consideration, difference 
array 704 may be subjected to a thresholding operation, wherein all values below a 
preselected value, for example 5 numeric counts in magnitude, are set to zero. Array 
706 illustrates the result of such a thresholding operation. The fact that all elements 
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of array 706 are zeros indicates that no significant changes occurred between 
preliminary photographs 701 and 702. 

A similar process reveals that between preliminary photographs 702 and 703, 
significant changes did occur at two pixel locations. Two pixels in difference array 
5 705 now have much higher numeric values, and those numeric values survive the 
thresholding operation as shown by array 707. Because it is unlikely that there are 
other intermittent sources of light in the scene, laser spot 504 can be confidently 
considered to be at the scene location corresponding to the significantly-changed 
pixels. The precise location in the camera's field of view may be determined by 

10 methods known in the art, such as by locating the largest change in pixel numeric 
value, or by finding the centroid of the pixels whose values changed significantly 
between consecutive photographs. 

Li one example embodiment, the size of selected photograph 601 may be 
specified in advance of taking any photographs. For example, reference photograph 

15 602 capturing the entire field of view 201 of camera 100 and using all of the pixels on 
the camera's electronic array light sensor may comprise 2,592 pixels width in the 
horizontal direction and 1,944 pixels height in the vertical direction, but the camera 
operator may specify, using controls provided on the camera, that selected 
photographs such as selected photograph 601 are to be taken with a size of 1024 

20 pixels width and 768 pixels height. These values are provided for illustration only; 
other sizes may be used within the scope of the appended claims. 

Figure 8 illustrates how camera 100 may select selected photograph 601 fi-om 
reference photograph 602. In Figure 8, the camera 100 has located laser spot 504 at 
pixel location (Xc, Yc). The width and height of selected photograph 601 have been 

25 specified to be W and H pixels, respectively. Given these parameters, camera 100 has 
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sufficient information to locate selected photograph 601 in reference photograph 602. 
Designating the upper left comer of selected photograph 601 as pixel location (Xo, 

Yo), 

W 

1) x,=x^-- 

5 and 

2) Y,=Y,-^ . 

If laser spot 504 is located near any edge of reference photograph 602, it may 
not be possible to position a selected photograph of a specified size in this way, as the 
boundaries of selected the photograph may extend outside the boundaries of reference 

10 photograph 602. In this case, camera 100 may position a selected photograph so that 
laser spot 504 is as nearly centered in the selected photograph as possible. 

Figure 9 shows a situation in which a selected photograph of dimensions W by 
H pixels cannot be centered about the laser spot location (Xc, Yc). The dashed line 
shows the boundaries of selected photograph 901 as computed by formulas 1) and 2) 

15 above. In this case, the camera may choose selected photograph 901 A by adjusting 
the position of the selected photograph so that it retains its specified size, but is fiiUy 
contained in reference photograph 602. 

As an alternative to adjusting the position of selected photograph 601 within 
reference photograph 602 when it is not possible to center a photograph of the 

20 specified size at the desired location, camera 100 may adjust the size of the 
photograph to be selected. For example, camera 100 may select the largest 
photograph that can be centered at the location of laser spot 504 while maintaining the 
aspect ratio of the photograph constant. 
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Figure 10 illustrates choosing the largest selected photograph that is centered 
on the location of laser spot 504. Laser spot 504 has been located at pixel location 
(Xc, Yc). Selected photograph boundary 1001 shows the location of the desired 
region. 

5 Additionally, maximum and minimum sizes for the selected photograph may 

optionally be specified. A complete example set of rules for choosing the width Wi 
and height Hi of selected photograph are given in the algorithm listing below. The 
desired aspect ratio (the ratio of the photograph's width to its height, typically about 
1.5) of the selected photograph is designated A, and the width and height of the 
10 reference photograph 602 are designated Wr and Hr respectively. The selected 
photograph may optionally have a minimum width Wmin and a maximum width Wmax- 



Listing 1. 



2 90 REM 

15 3 00 REM COMPUTE STARTING WIDTH AND HEIGHT, WITH OPTIONAL 

310 REM SETTING TO PRE -SELECTED MAXIMUM 

320 REM 

330 Wl=MIN(Wmax,2*Xc) 

340 REM 

20 350 REM COMPUTE CENTERING WIDTH AND HEIGHT WITH EDGE LIMITS 

3 60 REM 
370 H1=W1/A 

3 80 IF Xc<Wl/2 THEN 

390 Wl=2*Xc 

25 400 H1=W1/A 

410 END IF 

420 IF Yc<Hl/2 THEN 

430 Hl=2*yc 

440 W1=H1*A 

30 450 END IF 

460 IF (Xc+Wl/2) >Wr-l THEN 

470 Wl=2* (Wr-Xc-1) 

480 H1=W1/A 

490 END IF 

35 500 IF (Yc+Hl/2) >Hr-l THEN 

510 Hl=2* (Hr-Yc-1) 

520 W1=H1*A 

530 END IF 

540 X0=Xc-Wl/2 

40 550 Y0=Yc-Hl/2 

560 REM 
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570 REM OPTIONAL SETTING OF SIZE TO PRE-SELECTED MINIMUM AND 

5 80 REM ADJUSTING POSITION 

590 REM 

600 IP Wl<Wmin THEN 
5 610 Wl=Wmin 
620 H1=W1/A 

630 X0=Xc-Wl/2 
640 y0=Yc-Hl/2 

650 IF (Xc-Wl/2)<0 THEN X0=0 

10 660 IF (Xc+Wl/2) >Wr-l THEN X0=Wr-Wl-l 

670 IF (Yc-Hl/2)<0 THEN Y0=0 

680 IF (Yc+Hl/2) >Hr-l THEN Y0=Hr-Hl-l 

690 END IF 

15 Once this example algorithm has completed, a selected photograph location 



and size are determined such that the selected photograph is no larger than the 
predetermined maximum size, is no smaller than the predetermined minimum size, is 
completely contained within the reference photograph, is as nearly centered as 
possible on the location of laser spot 504, and has aspect ration A. The values Xo and 

20 Yo indicate the starting location of the selected photograph, and the values Wi and Hi 
indicate the width and height respectively of the selected photograph. Note that the 
selected photograph may be constrained to a fixed size by setting Wmax and Wmin 
equal to each other. Setting Wmin = 0 and Wmax = Wr configures the algorithm to find 
the largest selected photograph that can be centered on laser spot 504 within reference 

25 photograph 602. 

Once the size and location of the selected photograph have been determined, 
camera 100 can take a final photograph. A final photograph is the photograph that 
camera 100 has prepared to take. The preparations may involve preliminary 
photographs used for focusing, exposure determination, framing, or other purposes, as 

30 well as selecting a region to photograph. Photographing the selected region may 
involve taking a digital image of the entire field of view of the camera, and then 
extracting a subarray corresponding to the selected region fi-om the digital image for 
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storage. This is especially trae if the electronic array light sensor in digital camera 
100 is a charged coupled device (CCD) sensor or a complementary metal oxide 
semiconductor (CMOS) sensor. All pixels on the CCD or CMOS sensor, not just 
those in the selected region, may accumulate charge during the taking of the 
5 photograph, even though only those in the selected region will contribute to the final 
photograph. Digital camera 100 may measure the charges from all of the pixels on 
the electronic array light sensor and extract the final photograph from the resulting 
digital image, or may discard some or all of the unnecessary charges without 
measuring them. Whether accompUshed by any of these methods, the effective result 

10 is that the selected region is photographed. 

In one example embodiment, the laser pointer 104 may be interrupted so that it 
emits no light during the taking of a still photograph, and thus laser spot 504 does not 
appear obtrusively in the final photograph. 

Optionally, camera 100 may use the location of laser spot 504 as the center of 

15 a focus region, thus preferentially focusing on subjects in the vicinity of laser spot 
504. Typically, a digital camera performs focusing by adjusting a focus mechanism 
to maximize the image spatial contrast in a selected region of the camera's field of 
view. The focus region may be arbitrarily selected, but is often in the center of the 
camera's field of view. Selecting a focus region centered on laser spot 504 ensures 

20 that the portion of the scene that is of greatest interest, as indicated by the presence of 
laser spot 504, will be in focus. U.S. Patent 6,466,742, having a common assignee 
with the present application, describes a method of preferentially focusing at a 
designated scene location, and is hereby incorporated for all that it discloses. An 
advantage of the current invention is that the hght spot, such as laser spot 504, may 

25 itself add spatial contrast to the scene and facilitate focusing by camera 100. 



H. P. docket number 2003 11589-1 



Page 12 of 24 



In another example embodiment, camera 100 is capable of making video 
recordings. A video recording may be any sequence of successive digital images, 
sometimes called "video frames", captured at substantially regular intervals. The 
digital images need not be of a size similar to television video nor need they be taken 
5 at a frequency similar to television video. In a preferred configuration, laser pointer 
104 on remote control 103 flashes at a frequency of about one half the frequency of 
digital image capture during video recording. This arrangement ensures that most 
video frames will show a difference in the state of laser spot 504 as compared with the 
immediately preceding video frame. For example, if laser pointer 104 flashes at 

10 between 0.4 and 0.6 times the frequency of digital image capture, then at least 80 
percent of successive video frames will show a change in the state of laser spot 504 
from the previous frame. Camera 100 may adjust the composition of the video 
recording by re-selecting a region to photograph during recording as laser spot 504 
may move. In this way, camera 100 can simulate pan and tilt motions of a gimbal- 

1 5 mounted camera, but without the complexity of moving the camera. 

Unless precise synchronization is provided between the flashing of laser 
pointer 104 and the capture of video frames, laser spot 504 may appear in some video 
frames. In order to reduce the obtrusiveness of having laser spot 504 in the video 
sequence, automatic image processing using information from adjacent frames or 

20 adjacent pixels may be used remove the effect of laser spot 504. 

Figure 11 illustrates one simple example technique for removing laser spot 
504 from a video frame. Digital images 1101 and 1102 are consecutive frames from a 
video recording. As described previously, the presence of laser spot 504 has been 
detected in two pixels by computing an element-by-element difference frame 1103 

25 between the consecutive frames 1101 and 1102. Once laser spot 504 has been 
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located, its effect can be removed by copying pixel values from the most recent frame 
taken when laser pointer 104 was off. 

Other techniques may be envisioned for removing the effect of laser spot 504 
from video frames. For example, pixel information from both preceding and 
5 following frames could be combined to replace pixel data in a particular frame, for 
example by interpolation. Alternatively, the effect of laser spot 504 could be removed 
from a frame without reference to other frames, by replacing pixel data with 
information based on surrounding pixels. If the light emitted by light laser 104 is 
substantially monochromatic and camera 100 uses selective wavelength filtering on 

10 some pixels to generate color photographs, then light spot 504 may be detected by 
analyzing only those pixels that can sense the light wavelengths emitted by laser 104. 
For example, if laser 104 is emits red light, then it is likely that only the red-sensing 
pixels in the camera need be examined to detect the light spot 504, or need be 
adjusted to remove the effect of light spot 504 from a frame. 

15 In another example embodiment, photographer 102 may use motions of 

remote control 103 to communicate framing instructions to camera 100. For example, 
photographer may sweep light beam 503 over the scene in a rectangular, circular or 
other pattern that indicates a size of a region of interest. Camera 100 may detect the 
motion, and frame a photograph accordingly. 

20 Figure 12 illustrates one possible technique for selecting a region based on 

motions of the remote control. In Figure 12, a recorded set of locations 1201 
indicates where laser spot 504 was located in the previous 8 preliminary photographs 
taken with the laser 104 on. Camera 100 has chosen selected photograph 1202, which 
is large enough to encompass the entire set of laser spot locations 1201, and also has a 

25 typical photographic aspect ratio. 
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In another example embodiment, which may be combined with other example 
embodiments already described, camera 100 includes an optical zoom function, and 
uses its optical zoom capability to optimize photographic quaUty in some situations. 
In some cases, a selected photograph is defined that is completely contained within 
5 reference photograph 602 with excess area surrounding the selected photograph. That 
is, the selected photograph is not at the edge of reference photograph 602. Selected 
photographs 601 and 1201 in the Figures are of this kind, while selected photographs 
901A and 1001 are not. 

In this situation, camera 100 can improve the resolution at which it can 

10 photograph the selected region by activating its optical zoom function so that the 
camera's field of view just encompasses the selected region. That is, the focal length 
of the lens is increased, causing the camera's field of view to be narrowed, vmtil the 
selected photographic region is at the edge of the camera's field of view. 

Figure 13 illustrates using optical zoom to improve the resolution of a selected 

15 photograph. Selected photograph 1302 is entirely contained in reference photograph 
602, with excess area surrounding it. Camera 100 may actuate its optical zoom such 
that reference photograph 1301, rather than reference photograph 602, covers the 
entire electronic array light sensor in camera 100. Selected photograph 1302 can then 
be extracted from reference photograph 1301, but at higher resolution than if it had 

20 been extracted from reference photograph 602, 

In yet another example embodiment, light spot 504 may be used both for 
digital framing of photographs, and for controlling other functions of digital camera 
100. For example, laser 104, and thus light spot 504, may flash in a uniquely 
identifiable way (such as remaining on for three consecutive preliminary photographs 

25 or video frames, and then shutting off) to signal to the camera to take a final 
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photograph. SignaUng the camera to take a final photograph may also be called 
actuating the camera's shutter release. Using the same light source for digital framing 
and for controlling other camera functions saves the expense of having two different 
signaling methods. 

5 

The foregoing description of the present invention has been presented for 
purposes of illustration and description. It is not intended to be exhaustive or to limit 
the invention to the precise form disclosed, and other modifications and variations 
may be possible in light of the above teachings. The embodiment was chosen and 
10 described in order to best explain the principles of the invention and its practical 
application to thereby enable others skilled in the art to best utilize the invention in 
various embodiments and various modifications as are suited to the particular use 
contemplated. It is intended that the appended claims be construed to include other 
altemative embodiments of the invention except insofar as limited by the prior art. 
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